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ABSTRACT 

' In 1980, the Office of Juvenile Justice and 

Delinquency Prevention released a program announcement for an 
alternative education demonstration program intended to test a 
variety of school-based delinquency prevention programs. Great 
emphasi-s was placed on project evaluations. On? of the' programs 
identified was PATHE - Positive Action Through Holistic Evaluation. 
Evaluation of PATHE interventions relies totally on records kept at 
the schools, including student-level interventions, school and 
community level interventions. Intensity and fidelity rating scales 
developed for the evaluation of PATHE implementation provided useful, 
reliable and valid "data in a manageable form; The scores produced 
successfully compared levels of implementation of 2i different 
implementations in seven schools, and allowed for the testing of 
simple hypotheses regarding the correlates of successful 
implementation . Sample implementation rating forms, rating scales, 
and. program development worksheets are appended." (JAC) 
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In 1^80 f 'the Office of Juvenile Justice and Delinquency Prevention released 

a program announcement for an alternative education demonstration program. The 

program w^£ intended to test a variety of 'schocjl-based delinquency prevention 

models which would "impact upon the school climate, organizational structure, 
t 

and educational process" and which would "ultimately be adopted by school 

systems" '(p. 3) . In both the body of the program announcement and a lengthy 

appended background paper, great emphasis was placed on the importance of the 

•* » 

thorough evaluation of funded projects. In a section delineating standards 

* * 

for evaluation, the documentation of program activities and of the duration 
and intensity of services was included with the comment that"without documenta- 
tion of these program elements, outcome evaluation studies are relatively use- 
less for policy making since they do not describe what generated observed 
results, making replication impossible" (p. 38) . 

The importance of describing program processes has been recognized in a 
number of evaluation models^ (Stake, 1967; stuff lebeam, et al., 1971; Provus, 
1971) . Morris and Fitz-Gibbgn (1978) discriminate between summative jand forma- 
tive purposes for the 'evaluation of implementation. Program documentation serves 
the summative purposes of providing accountability data, describing the program 
under evaluation, and identifying possible causes of program effects. Formative 
purposes are served by program monitoring including the updating and revision of 
* program plans (p. 15-22) . As operationalized by the national evaluation team 
contracted by O.J.J. D-P . to evaluate the alternative education program,^* both 
summative and formative purposes are served in ,the evaluation of implementa- 
tion . 

- The purpose of this paper is to describe the evaluation of the implementa- 
tion of PATHE - Positive Action Through Holistic Education - one of 18 projects 

"*"The national evaluation team, headed by Drs. Gary and Denise Gottfredson, is 
based at the Center for the Social Organization of 'Schools at The Johns 
Hopkins University. 



funded by O.J.J. D.P. Specifically, the process used to collect and analyze 
* implementation data and some initial findings regarding the implementation of 
a juvenile justice delinquency program in the public schools will be discussed. 

PROGRAM DESCRIPTION • • 

Context and Staff ' 

PATHE operates in seven Charleston County (S.C.) schools, four middle 
schools and three high schools. Three middfe and two high schools are legated 
in downtown Charleston and are almost 100 per cent blac*. One middle and one 
high school are located on John's Island and serve an integrated, rural popu- 
lation. At each of these, schools, Cwo full-time staff members a Student 
Concerns Specialist (S.C.S.) and a Curriculum Specialist (C.S.) - are, responsi- 
ble for carrying out PATHE activities. At the central staff level, PATHE em- 
ploys a consultant and an evaluator who manage and monitor program ,implementa- 
tion. The Charleston "Count* School District Federal Programs Director, author 
of' the PATHE proposal for funding, has continued to be actively involved in the 
implementation of the program. ,f 
Goals and Interventions 

• As its name' implies, PATHE is designed to impact upon the whole child and ■ 
the whole school environment. Therefore, services are not limited to individual 
students; rather, PATHE also attempts to affect changes in the school which will 
improve. conditioner the entire student body. Consequently, although many of 
PATHE 's interventions are d/rected toward providing supplementary affective and 
academic services to studeits, others are intended to build faculty cohesion, to 
improve school climate, a/d to bring about organizational changes. • The stated 
goals of PATHE are 1) td reduce' the occurrence of delinquent acts and inappro- 
priate behavior in cfnd 'around PATHE ' school s , 2) to reduce unexcused absences 
and tardies, 3) to increase successful transition to the job force 'and post- 
, secondary education, 4nd 4) to reduce academic failure. 



Through a process designed and facilitated by the national evaluation team, 

V 

a Program Development Worksheet (P.D.WO has been developed which describes the 
program's objectives and interventions in detail* and blinks them to these goals. 
In addition, jthe strategies for implementing each intervention - including tasks, 
couple tion date^, and persons responsible - are outlined. The P.D.W. is updated 
twice a month and revised as necessary and provides a continuous history of in- 

9 

tended program activities. It also documents progress made toward implementation 

i 

and obstacles encountered. . „ * 

Target student sele'ct ion and treatment. ' a group of approximately 100 stu- 
dents at each school were selected to receive intensive services from the PATHE 
spedialists^ During the 1980-81 school year, target students were selected 
based on teacher referrals and schpol records before the evaluation had been 
designed. However, a more systematic selection process was implemented for the 
current school^ year. A pool of students in need of services was identified at 
each school using achievement te^t (C.T.B.S.) scores, attendance and suspension 
♦records, teacher* and specialist referrals, and classroom grades. Members of 

the pool were randomly assigned to treatment (target) and control groups., 

* ■* « * 

Most of PATHE 1 s services are available to all students in the school. How- 

ever, target students receive the following services not offered to others: 

m 

1) The specialists systematically collect information about target 
students and maintain^ "Student Profile." Information collected 
•includes detailed analysis of individual CTBS results, past class- 
room grades, attendance and suspension data, discipline referrals, 
and teacher comments. * •'.<•' 

2) These data are used to prepare problem statements and/br instructional 

needs assessments £nd to _ plan, appropriate interventions for each stu- 

dent. Strategies include 4 tutoring, counseling, peer counseling, 

leadership training, involvement in extracurricular activities, 

j 
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and self -improvement contracts. 
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3) Specialists conduct at least one counseling session per month with 
every target student for the purpose of monitoring progress and re- 
vising individual plans as needed. 

♦ 

4) An effort: is made to link target students to all PATHE and school 
activities. 

5) The parents of target students are periodically contacted by the 

4 

specialists. 

Classification, of interventions. It has been the nature of the PATHE pro- 
gram to be extremely responsive to opportunities and needs which arise as the 
program progresses. For this reason , interventions have been t deleted, added, 

„ 4 

and modified as necessary. Figure 1 presents a list and brief description of 
ongoing^interventions classified as follows: } 

. 1) student-level: interventions which provide direct services to individ- 
ual students; ' . 

2) school-level: interventions (directed toward the expansion* and improve 1 - 
ment of existing school services and the' introductiolf of new structures 
and organizations within th£ school; and 

3) community-level: interventicpns which develop linkages between the 
school and community support systems.^ * 

Decisions to classify interventions using these categories are not always clear^ 
cut.. For example/ the tutoring program includes a strategy to involve tutors 
from institutions and agencies outside the school. However, since the central 
purpose of the strategy is to serve individual students, it is classified as a 
student-level rather them a community-level intervention . » Similarly", the peer 

counseling program serves individual students but the thrii^t^ of PATHE activities 

# ■» 

related to the intervention has been to train a group of students to serve as 
counselors and set up a schedule for regular counseling to occur. . Therefore, 
peer counseling has jbeen classified $s a school-level iniervention. 



Figure 1. Classification and definition of PATHE interventions. 



*Studenfe-Level Interventions 



Tutoring Tutoring is provided to students by the Curriculum* Specialist , peer 
tutors, and tutors brought into the school from outside agencies. 
Students who receive tutoring may be self-referred or referred/ by 
teachers. In addition to tutoring provided outside of the classroom, 
the Curriculum Specialist may providfe special materials^ and instruct 
tional plans to the students' classroom teachers. . , % 

Counseling The Student Concerns Specialist provides counseling to students, ^ 
who are referred by teachers 1 and administrators and who are absent, 
tardy, cut class, or are suspended. In addition, both PATHE 
'specialists conduct regular counseling sessions with target students. 

Study The Curriculum Specialist provides training to students in the form 

Skills of mini-courses on study skills (p.g., note-taking, listening skills, 

good study habits) ... 



Job- 
Seeking 
Skills 

Services 
to Target 
Students 



The Curriculum Specialist provides ftraining in spedific skills related 
to finding and keeping^a jab (e.g., interview etiquette) and offers 
opportunities to broaden .career awareness. 

Students identified' as targe.t (approximately 100 in each school) 
receive special services including diagnosis of achievement weaknesses 
and individual remediation plans, additional counseling, linkage to ' 
air PATHE and school activities, and regular monitoring of progress 
toward behavioral and academic goals. 



School-Level In€erventions 



Resource 
Room 



Faculty 
Inservices 



School 

Pride 

Campaign 

Curriculum 
Review and 
Revision 

Field 

Trip 

Program 



The Curriculum Spjgc^ialist sets up and monitors the use of a PATHE 
Resource Room. Jjfesources include self-instructional activities, 
books and magazines for free reading, and other supplementary 
instructional njaterials. Both teachers and students are encouraged 
to use the resources' provided. . 

PATHE works with the teachers and principals to identify training 
needs at each school and to provide additional resourcs for faculty 
training, especially in areas<*i^evant to PATHE 1 s goals. Inseirvice , 
topics have included ciassroom management, Student Team Learning, 
and faculty team building. 

With the cooperation of Parent and Student Leadership Team member^, 
the Student Concerns Specialist plans and implements activities, to 
.improve school -pride . ^ 

Achievement test results are used to diagnose school-wide academic 
weaknesses. The Curriculum Specialist, in- cooperation with the 
Curriculum Support Team, uses the resulting information to plan and 
carry out, remedial programs. 

PATHE provides additional resources to the schools to assist with 
fiett^J^flps which support PATHE 1 s goals and objectives. PATHE staff 
members may conduct field trips themselves or assist other teachers ♦ 
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Reading A period of time is set aside in the school schedule for free reading 
Experience for everyorte in the building. Teachers, custodians, and adrtiinistjtra- 
Program tors as well as students are encouraged to participate and students 
^.are rewarded for active participation. 



k .Exploratory 
Program 

i 

Discipline 
Peviev and 
Revision 



Peer 

Counseling 



In the middle schools , PATHE works with the principal and the fadulty * 
to provide meaningful exploratory activities as mandated in the S.C. 
minimum required program. 

The Student Concerns Specialist reviews information about discipline 
problems in the school and plans and carries out activities designed 
to address these problems/ Emphasis is placed on student involvement 
in the development of school, and classroom rules and the establishment 
of a discipline referral procedure and the use Qf a standardized 
discipline referral form* 

Students are selected and trained to conduct rap sessions with other 
students in the schooi. The purpose of peer counseling is not to 
provide personal counseling, but rather to conduct a forum on' topics 
of special concern to students and to establish peer pressure to deal 
with problem areas in a socially acceptable way* 



Expanded 
Extra- 
curricular 
Activities 

Student 

Leadership 

Team 

Student 
Concerns 
"Support^ 
Team ' 

Curriculum 
\Support 
Team 



Career 
Explora- 
tion 
Programs 



Business 
Education 
Partner- 
ship j 

Parent 

Leadership 

Team 



The Student Concerns Sepcialist encourages the growth of** extracurri- 
cular activities on campus by assessing student needs, recruiting 
sponsors, ^nd monitoring club progress. 

At each school a group of student^ is Actively involved in planning , 
ways to improve their school and in implementing their plants. These 
students also receive leadership training. 

Five faculty members at each school work" with the S udent Concerns 
Specialist to pl&n activities which will improve school climate and 
the behavior of individual students. 

Another group 4 of five faculty members works with the Curriculum 
Specialist " to pl^n and implement activities designed to impirove 
academic performance at the school. 

Community-Level interventions - ~ < ^^\ • 

In cooperation with Trident Technical Cqllege, student's at PATHE "igh^ 
schools participate in two programs designed to introduce" them to 
.careers in engineering and industrial* technology. During the "school 
year, students of bath sexes attend six. career awareness .sessions at 
the college* During the summer, young women attend 24 sessions of the 
FACET (Female Access to » Careers in Engineering Technology)* program 
for which they receive two college credits. 1 

In cooperation with the school district's joint program with the 
Chamber of Commerce, PATHE works to establish an^activ^, productive ' 
partnership for each of its schools with a business in the community. 
The. primary purpose tff, these partnerships is to provide management ^ 
and public relations expertise to schools. 

Similar to the student teams, most PATHE parent groups plan ways tg 
improve parental involvement .in the schools. These teams also receive 
leadership training. 
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Figure 1. Classification and definition of PATHE interventions. 
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• . * METHODS 

i . ■ 

■ • * i 

Design I ssues 

~~ ' - L • 

Level of detail* PATHE is a- large program, both in terms the number of 

schools, students, and teachers served and of the number and convexity o'f in- 

terventions to be implemented- As Morris and Fitz-Gibbon (1978) point out, 
* » • 

the appropriate, level 6f detail" for descriptions qf- program components is de- 

termined by the level of precision with- which program developers have described ' 

i 

the intended interventions. In the case of PATHE, the P.D.W. provide? an ex- 
tremely detailed blueprint for each intervention necessitating an equally 'de- x 
tailed monitoring plan. The implementation for any single intervention varies 
from school to school and when diffe^nces in implementation between schools 
are considered acros 21 interventions, the resulting flood of information t;an 
cleave the evaluatbr floundering. The greatest challenge to the evaluator be- 
comes to find a way to both describe the program at each site -in adequate detail 
and to summarise implementation data in a useable, comprehensible, valid >toay: 

According to Gephart (as cited by Reeves, 1979) when ^evaluating the impJLe- c 
mentation of a complex *inn ovation (such as PATHE) it becomes very *dlf f icult to 
arrive at a single score which accurately' portrays implementation. Gephart- 
recommends tfre use of a profile of scores instead (p. 5,) 4 . 

In' the 'evaluation of PATHE an attempt >as been made tcfl v offer the best of % 

both detailed and summary information. Detailed desCription^'of interventions as^ 
• i • • 

implemented at each school are prepared twice a year for use by program monitors 

and the national evaluators. These rfeports include a description of the inter- 

vent ion as planned, a number of indicators of implementation, a profile of 
• * T • 

implementation scores, and verbal descriptions of modifications and expansions 
of the intended intervention. In addition., a single implementation score for 

each intervention and for the whole PATHE program at each school is provided. 

* * * r 

These scores are generated using the rating scales described in detail below. 

, > 

S . 1 



Level of participation. , Rossi, Freeman,, and Wright (1979) stress the impor- 
tance of measuring the extent to which members of the target population actually 
.participate in program activities (p. 123-126) . Indeed, the intensity of PATHE 
^services - the, actual number of incidents of contact with students identified as 

target - has been a maj<3£ concern of program managers. Students who receive 
♦ 

PAXHE service.s are not adaptive audience. The specialists do not have a 

classroom of students to whom they can address their services nor are students 

systematically scheduled to Attend PATHE activities aj| part of their normal 

school day. If services are in fact -to be delivered, the specialist must be 

i ■ *, 

proactive in seeking out target students, scheduling them for services, and 
Motivating them to attend. In drder to adequately monitor the student -level 
interventions, data collection instruments have betenc developed to ensure that^ 
contacts -with student^ arg accurately recorded. 

Data collection methods* Three methods of collecting implementation data' 

: * / 

are commonly suggested in the literature: use of existing or program-specific 
recdrds, direct observation, and self -reports of program personnel or partici- 
pants (Morris & Fitz-kibbon, 1978, p. 51; Rossi, Freeman, & Wright, 1979, p, 126- 
144; Wolf, 1979, p. 83-88) * PATHE program interventions in general - and 
especially those directed toward individual students - do not lend themselves to 
direct observation. It would be very difficult to observe i personal counseling . 
session in an unobtrusive and yet ethical manner . ( Adequate sampling for the 
observation of less sensitive interventions (e.g., team meetings) at seven 
• different sites would place an unrealistic burden on extremely limited evaluation 
resources. Consequently, the evaluation of PATHE interventions has relied almost 
totally on record? kept in the schools, program manager irecords, and periodic 
interviews and other self-report measures administered to PATHE staff members 
and principals. t 
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Data Collection 

, student- level interventions,' Daily Contact .Logs and Activity sign-ins are 

" - y 

used by the specialists to* record contacts wit±«*idividual students and groups 
of students. . In the development of these instruments , great care was taken to 

/ 

ensure that a nvinimum amount of time would be required to record each contact. 
\ * 

■ More complicated procedures used during the 1980-81 school year resulted in 
incomplete and inaccurate records of student-leyel services, pri m a r ily because 
the specialists understandably preferred to devote their time to the students 

V 

rather thc&i to the completion of forms. Data collected on these forms include 
the nature or purpose and duration in, minutes of each contact and tfie presence 

of other persons (e.g., teachers, parents, administrators)* 

• * * 

School district suspension , expulsion, and dropout data are also collected 

' ae well as discipline referral forms. Information* from suspension rejbort^, 

Daily Contact Logs, and Activity Sign-ins are entered on local 'computer „files^ 

from which summary report^ are prepared for use in monitoring the* delivery of 

services. Furthermore, Student Data Sheets are produced which compile all data 

collected for each target itu<Jent. These reports are 'distributed to the special- 

V • 

ists and added to school files as a case history of each student's contact with 
the program. Discipline referral forms are sent to the national evaluation 
team' for analysis antf the resulting reports are shared with the program .staff 
on a regular* basis. 

/ Student Profiles' are maintained in the schools which include records of 

parent contacts, individual instructional plans-, _and behavioral problem state- 

• ments for each target student. The 'status of the files - what plans are on 

« 

^ file and the quality of these plans^- is periodically checked as an indicator of 
v * the -quality of services being provided target students. 

School-level interventions . „ Faculty Contact Logs, similar to the Daily Contact' 
Log used with students, are maintained 'by, the specialists as a record of their 

ERIC ' 1.1 . 



contacts with individual teachers. Activity Sign-ins are also used with groups 
of teachers such as the support teams'. Resource room sign-ou£ records, are main- 
tained and turned in on a monthly basis as a form of documentation of the use of 
PAT HE materials. Specialists also submit action 1 plans, agendas, and minutes 
from meetings conducted~at their schools (e.g., Student Leadership Team meetings) . 

Qnce a month, the evaluator reviews the P.D.W. and compiles a task calendar 
which is distributed to the, staff meters. Specialists use the calendars to 

4 

T 

plan their daily activities, indicate completion dates of tasks assigned, describe 
problems or successes related to each task on -their copy, .and return the calendar 
at the end of 'the month* The completed calendars serve as^means of communica- 
tion ^between the specialists and the program consultant who can use them -to plan ^ 
her monitoring visits.. They are also kept on file as one form of self-report 
.implementation data. * * 

Occasionally specialists are also assigned the task of completing an 'Activity 
Report describing the implementation status of a specific intervention at their , 
school. ' 

Community-level interventions. Activity sign-in's, plans, minutes, agendas, 
activity reports, and monthly calendar task assignments are also used to collect 
data abput community -level interventions. .In. addition, in cases where a program 
is managed prijo^rily by the PATHE program consultant when, data are collected ^ 
away f^om the schools, special .forms are developed, used, and collected. For 
example, instructors at 'Trident. Technical College collected attendance data for 
PATHE students participating in their Career Exploration program. 

* 4 

Data Analysis % 

. implementation reports. Twice a year - at t^he end of the. first semester and 
in the late spring - implementation reports fare*, prepared £or each school. The ^ 
report consists o.f a separate form for each intervention. A description of the 
intended intervention taken €rOm the -p.D.W. is presented on one side of the form 
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with space available on the otheApide for a description of what has actually 
taken place at the. school. All .information relevant *t5*the intervention at,. a 
single school is compiled and presented on the form" including summaries of team 
plans, frequencies of student contacts for a specific purpose', and other evi- 

■i 

dence as »to whether or not the intervention has^be^n implemented and to what 
* degree* A draft report is prepar'ed and used as a guide for school -by-school * 
Staf ^interviews conducted by the program consultant and/or, .the evaluator. 
The purpose of the interviews is to verify the accuracy of the information on the 

" * * » 0 

report , collect additional information where data on file do not adequately 
^ ' portray programs at *£he school', identify problem areas,, and develop strategies . 
to" improve the program ^ After the interview, the reports, are revised and up- 
dated as* necessary. (Appendix ^presents documents wtfich follow the monitoring 
of one < intervention from the P.D.'W. strategy description, to the monthly calendar , 
to completed implementation report fpr one school.) 

Rating scales. Three rating scales -* of fidelity, intensity, and duration w 
have been developed for use in reducing the large quantities of data collected 
to numerical measures of implementation. In addition a scale ha^ been developed 
to rate principal support ot each intervention and another to rate the level of 
central staff monitoring of intervention implementation. 

Fullan and Pomfret (1975, 1977) in their review of implementation . studies 
identified'a "fidelity" approach to the measurement of implementation. Fidelity 

can be defined as the degree^ to which a program as implemented corresponds to 

s t * „ 

the expectatations of the program developer. Morris and Fitz-Gibbon (1978) 

suggest that such an index can be particularly useful if there is a detailed 

program plan from which to draw the intended characteristics of the intervention 

(p." 75)1 The P.D.W. makes the delineation of the critical characteristics of 

\ each P^THE intervention a relatively simple task. 

Typically, PATHE interventions, describe a series of tasks or a process 
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which is to be carried out at. each school. For example, the Student Leadership . 
Teams are expected to develop a plan of action to improve school climate . The 
plans may be quite different in each school but a plan should, exist . The tasks 
or activities identified in the p.D.W. "and listed on the implementation report, 
'described above are also 'listed on the fidelity rating instrument as standards. 
(See Append^ix B -for copies of fidelity and intensity scales) . By referring to the 
evidence of implementation reported on the .implementation report, the rater can 
fairly easily check off those tasks which have in fact been carried out. Once 

checklist is complete^, the following scale is used to rate implementation: 
4=exceeds standards; 3=meets standards completely or with approved adaptation, 
2=^e*ets 50-99%* of standards, l=meets 1-49% of standards, 0=does not meet 
standards/not implemented. . » , 

The intensity scale also describes program manager expectations or standards 
for the number of persons served or activities conducted for each intervention. 
Again, the information necessary to assess to what degree the standards have 
been met at a school is available on the implementation report. The following 
* five-point intensity scale is, Used to assign ratings: 4=exceeds standard by more 
than 5%, 3=meets standard + 5%, 2=meets 50-95% of standard, l=meets 5-49% of 
standard, Ofmeets 4% or less of standard. 

' .The dui ration .scale simply asks the rater 'to code the date of initiation of the 
intervention using numbers compatible with the fidelity and intensity ratings as 
follows: |=in operation August-September, 1981, 3=began in October, 2=began. in 
November, | l=began in December/January, 0=had not begun by end of January. Dec- 
ember and January are collapsed into one rating on this scale because of the 
. Christmas holidays which effectively cut program time in half during those 
months.. 

Reliability coefficients for each subscale were calculated us^ng a 
Cronbach 'c When ratings are used to produce a school-level .rating across all 

14 
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■ • * 

^,.21 interventions, the fidelity scale was found to have a reliability CD-" - ' 

efficient of ,67, the intensity scale a coefficient of -79, and the duration 

* * * * 

scale a coefficient of .29. Due to the lack of internal consistency demonstra- 

ted by the duration scale, resulting data were not used in further analyses. 
An Impleuientation Scale which combines the fidelity and intensity ratings demon- 
strates adequate internal consistency with a reliability coefficient of .89. 

' In contrast, when fidelity and intensity- ratings are used to produce an 
.implementation score for each intervention across schools, the internal con- 
sistency of the scales drops to .16. Since only seven schools are involved in 
the program, this drop in reliability is not surprising. Furthermore, since 'we 
would expect, the level of implementation on each intervention ^to vary, often, 1 
dramatically, from school to school, # the internal consistency of the scale when 
used to rate .interventions is understandably low. - To date only one rater - 
the program e valuator ' - has been involved in the use of the scales. In the future, \ 
it 'may be possible to use 'multiple raters and establish inter-rater reliability 
for intervention ratings. However, the "'logistical difficulty of- finding several 
people willing to spend the time necessary to rate all interventions on two 
scales for seven schools makes it unlikely that an opportunity to assess inter- 
rater reliability will arise. Furthermore, the standards on the - rating scales 
" will be different every time the. scales are used since the program will have _ 
progressed through another half year of implementation and expectations will 
have changed accordingly. Consequently, the cost effectiveness of establishing 
inter-rater reliability for one iteration of : . the scale is questionabie ; 

Morris and 'Fitz-Gibbon (1978T , recognizing the problem of establishing 
reliability for implementation measures suggest, that evaluators focus on demon- 
strating the validity of their instruments (p. 135) . They describe concern 
for validity of implementation measures in terms of a four-part question: 

"Is the description of the- program which the instrument presents accurate , 
* ' ' ' 
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, relevant, representative, and complete?" The instruments developed to measure 

* » 

♦ * m 

the implementation of PAlflE meet all four requirements. ^ The fact that ratings 
are based on specific standards taken directly from the Program Development Work- 
sheet - a document prepared by program developers - is evidence that the scales 
are relevant , representative , and complete. Since the rating process requires 

the rater to review data collected from multiple sources and compare the evidence 

• * 
of implementation to an objective standard, the accuracy or absence o£ bias. of. 

. ratings i*s established. Furthermore , the presentation of all* ratings, evidence 

of imglement/ation, and standards on the implementation reports makes the derivatidn 

* r ■ ■ V - ' . : " 

of the ratings a process open to public review and challenge. 

The principal support rating scale is not as easily defended, primarily * 
because evidence of principal support is often not as clear-cut as indicators of 
program implementation. Principals were rated on their support of ~e^ich intervention 

.by the evaluator and ttoo program managers using the following sc&le: 4=principa!l 

• * * * * • r - . 

has done something to support the intervention and gives enthusiastic verbal support, 

3==the principal expresses enthusiastic verbal support not contradicted by actions, 

2=the principal supports moderately, with some reservations or concerns, l=lfhe 

principal neither* supports nor obstructs implementation, A=the principal obstructs 

implementation. * K * ^ 

Each rater completed the scal6 independently. The eyaluator used principal 

intervidV responses concerning each intervention and evidd$^e on the implementation 

reports to assign ratings. For example, in several schools the specialists reported 

* 4 / 

that their principals had conducted a faculty meeting about grading practices using , 

information cpllected by PATHE, These^ reports were considered evidence that 

♦ / 

principals had done something in support o£ the Curriculum 4 Review and Revision 

• • • < *< * - 

intfdrvention resulting in a principal support rating of 4. The other two raters , 
* drew on their own experiences in # implementing the .program to ratte principal 
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support • 

Once all ratings were completed, the evaluator compared the ratings and made 

4 * 

a finals decision on the score for each intervention. It was felt that a mean score 
would be inappropriate since each rater was operating out of a separate set of 
exgeriences. Rather, a set of decision cpridelines were used to detern&ne each 
rating* If one rater had assigned a score of 4 to an intervention and both 
of the <Jther raters had assigned 3's, the evaluator assumed that the person giving 
_a 4 rating had information or evidence that the principal had done something to 
-support the intervention which was unavailable to the .other two raters. Similarly, 

if one rater had given a rating of 0 and both other ratings were low, the evaluator 

> 

assume^ that the rating of. 0 was based on information about principal obstruction 
of In intervention which the .other raters did not possess. For ratings in the 1 
"\£o 3 range, either a mid-score or the score given by two of the raters was used. 
When principal support scores for all interventions attach school were calculated 
using the resulting ratings, th# internal consistency of the support rating scale 



using Cronbach c< was .86'. • 1 

A final scale was developed to rate the level of central staff monitoring 

and involvement in the implementation of each intervention. In developing the 

scale, two types of evidence of monitoring and involvement were considered. First, 

* ** 

a review of monthly calendars was conducted ^6 determine how iiiany times during the 
schodl year the specialists had been required to turn in a document of some kind 
(e.g., a plan, a report, data) for each intervention. When turn-ins are required* 
those who fail to respond are periodically reminded of the deficiency until it 
is corrected, resulting in extensive monitoring. Activities for which no tiirn-in 
are required are left to the conscientiousness of the specialist and less 
systematic monitoring by central staff personnel. It was that the number 
of turn-ins ranged from zero to eleven per intervention with a mode of* 3, a mean 
of 4, and a standard deviation of 2.7.« „ 
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A seconjd consideration in rating monitoring was whether or not activities 
had been conducted or action taken from the central staff level to facilitate 
intervention implement at ion. For example, at the first of the school year, the 
program monitor visited each principal t& persuade him to use a new discipline 
referral form developed' for PATHE* In the case of another intervention, a 
project-wide 'conference was coriSQcted for members of the Student .Concerns and 
Curriculum Support Teams to help te!am members develop action plans for the year. 
The following & ^' e was used to rate the degree of monitoring: 4=centrally 
organized' activity and 3 or ^ore turn-ins required on calendar, 3=centrally 
organized activity or 6 or more turn-ins required on calendar, 2f=l-5 turn-ins~ 
required on calendar, l=no turn-ins required. 



RESULTS 



Implementation Scores 

Average scores were calculated on both the fidelity and intensity scales for 
eacb school' and each intervention, Cn the fidelity scale, school scores ranged * 
from 1;85 to 2 ; 95 and intervention scores ranged from 1 (Job-seeking Skills) to 
3.28 (CurricUlqm Review and Revision). " The mean score for ail schools and* all 
interventions" was 2.2^with a standard deviation of 1.069. 

On the intensity .scale, school" scores ranged £rom 1.85 to 3,6 and intervention 

scores ranged form 1.14 (Services to Target Students and Business Education Partner 

ship) to 4 (Student Leadership Team) . The mean intensity score for all schools 

and all interventions was 2,34 with a standard deviation of 1.467. It should be 

remembered in interpreting intensity scores that 4 the rating process compared the 

number of students served to a standard for the intervention and the resulting , 

scores do not represent absolute numbers of students served. Therefore, it may 

be 'that, even though the intensity rating is lower for Services to Target Students, 

there h^ve b^en more contacts with target students than with members of Student 

* 

Leadership Teams. , . 

18' 
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Using combined fidelity and intensity ratings, the student-level interventions 
as a group had a mean implemntation score of 1.87 (standard deviation=1.297) , the 
school-level interventions had a score of 2.522 (standard deviation=1.205)-, and 
the community-level interventions had a score of 2.176 (standard deviati"on=1.405) . 
These ratings indicate that the program managers 1 concern that services to indi- 
vidual students are not being implemented as intended' is indeed warranted. Perhaps 
of even greater concern is the fact that the intervention Services to Target 

Students' received a mean fidelity rating of 1.28 (standard deviation=.487) and 
* # 

a mean intensity rating of 1.14 (standard deviations 690) . .If a rating of 3 is 

interpreted as an indicator of /full implementation as described in the.P.D.W., 

/ * 
the student-level interventions as a group and the Services to Target Students - 

intervention fall well below the standard. On the bright side, the following 

interventions received meem ^scores of *2.5 or above: Study Skills, Resource Room, , 

School Pride- Camtoaigii, XXi^riculum Review and Revision, Sfijdent Leadership Team, 

Curriculum Support Team, Career Exploration Program, and Parent Leadership Team. 

Mean princ/pali support -scores for school ranged form 2.4 to 3.7 wh>4e 

£ i 1 * \ 

support scores fcpjT interventions ranged from 1.42 (Job-Seeking Skills) to 4 
(Student Leadership Teams). The average rating across all schools and intervene 
tions was 2.94 with a standard deviation/ of 1.001. * * . . < ' • ■ 

Lipham (1977) reports that "it is -lively that no m^jor program .of eduGationaJ- 
improvement can succeed V/ithout the understanding, support, and involvement 6f 
the administrator of*thfe. local school" (-p. 119) and, Berman and jfcmu^hlin (1977) 
found that "if the innovation is complex the' blessing of the ^principal and^active . 
support are. even more important" jE t JL28) . JOien principal suppbrt r^tin^s were, 
correlated with implementation scores |or each intervention, a Pearson correlation 
coefficient of .66 was obtained" (r 2 =.^). While it is not possible to conclude 
based on these data that principal support of an intervention causes it to be 
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mpre fully implemented, principal support may well be a critical facilitating 

element if not an essential precondition to complete implementation. 

Monitoring scores' were also correlated with implementation scores for each 

2 

intervention and found to yield a coefficient of. 54 (r =.29). Although this 
relationship is weaker than that between principal support and impleit^ntation, 
it is s^ill of^sufficient strength tp be useful in understanding difference s *in 
implementation. Especially in a program such as PATHE which requires so much 
of staff members, there is a natural tendancy to do first the tasks which are 
being most closely observed by supervisors or for which^a concrete product must 
be produced. It is particularly interesting to note that the two interventions 
which received the lowest monitoring scores - Job-seeking Skills and Services to 
Target Students - also received the lowest -implementation ratings. The ''Job- 
seeking Skills- intervention has never been a high priority for central staff 
m£mbei;s and the vratings it received are to be expected. However, Servijces to 
Target Student's have been stressed repeatedly in staff meetings. U^f ortunately , 
the forms used to dociiment services to target, students (e.g., problem statements, 
t * records of parent contacts, instructional plans - are not turned in but are kept 

\ y . 

at the school in the Stud&it Profiles . • 

» DISCUSSION . ( 

Rating Scales 

The intensity and fidelity rating scales developed for the evaluation of 
s PATHE implementation have provided useful, reliable, and valid data in a manageabl 
form. T^e scores produced using the rating scales were succe'ssfully used to 
Compare levels of implementation of 21 different interventions in seven schools 
and also allcSw the testing of simple hypotheses regarding the correlates » of 
successful implementation. 

» * An assumption upon which the scales are based and which might be questioned / 

^ » * 

is that the multiple standards specified for each intervention, on the fidelity 
q scale are of equal importance and that fidelity and intensity ratings aire also 

; Bye ' 1 
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equally important indicators of implementation. If an argument can be successfully 
made that this assumption is false, it may be necessary to weight ratings of com- 
ponents- judged more important. 

Another underlying premise on which this entire data collection effort has 
been based is that program personnel and the evaluator interact in a trusting J 

relationship. Fortunately for this evaluator, the two people most directly respon- 
t 

sible for the implementation of PATHE are enthusiastic and informed consumers of 
evaluatioj*' results who see the program development process as an asset and rely 
on both the local and .the national evaluator s for inf ormation which will help them 
improve their program. It is truly difficult to imagine how the detailed da€a 
required to produce the implementation reports on which the rating scales are based 
, cou^d possibly be collected without the unreserved support of program staff members. 
The rating scales were developed for use in the formative evaluation of PATHE. 
For better or for worse, circumstances now jexist which will probably lead to the 
summative use of fidelity and intensity scores. Cut backs in O.J.J.D^P.'s funding 
will trickle down* to PftTHE and prograin managers 'are currently trying to decide which 
of the seven PATHE schools will not continue in the program next year. Since outcome 



data will not be available in time to inform this decision, implementation data 
will be used to select PATHE schools for 1982-83. The systematic collection and 
reporting of implementation data which has characterized the evaluafc±tfn of PA^THE 



\ | will allow program managers to feel confident that their decisions are as fair as 



w 

\ 

^ possible. 

V 

Implementation of PATHE 

Berman and McLaughlin (1977) hypothesized that ^complex and ambitious innova- 
tions are more likely to elicit the enthusiasm of teachers than routine projects" 
(p. 82). PATHE certainly qualifies as a complex and ambitious innovation and it 
has elicited enthusiasm on the part of teachers, principals, students, and staff 
members. It has also elicited frustration, exhaustion, and some very good humor. 

9 21 ' 
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PATHE is now approaching the end of its second year of implementation. Some 
of its interventions are doing, very well. While we still do not )oiow what impact 
each intervention is having on student outcomes, we at least know that many of 
PATHE 1 s intended activities are in fact taking place and have received the support 
of^the principals involved. However/ othfer interventions are still marginally 
implemented. As PATHE 1 s managers decide whether or not to continue the program 
in weak schools, . t.hey might also consider whether or not to continue weak inter- 
v&ntions. t *hiree ' factors should be included in such a decision. First of all f 
a Judgement should be made as to the critic^lity of the intervention to the goals 
of PATHE and the theory which supports its program plan. Next, a review of 
related interventions should be made to determine if another, more successful 
intervention addresses the same goal. Finally, \t woul&be wisfe to check principal 
support for the intervention at each school. Us*in§ these\criteria, the Job- ^ , 
seeking Skills intervention would probably be dropped from PATHE during the next 
school year. While it is directly relate^ to the program goal of improving student 
transition to the work force., the Career Exploration Program at Trident Technical 
College is a more successful intervention related to the same goal. Furthermore, 
the Job-seeking Skills intervention does not enjoy a^igh level of principal 
support. In contrast, 'the Services to Target Students Intervention is strongly 
supported by five of the, seven principals and is trie only intervention which 
attempts to provide intensive treatment to students identified as exhibiting patterns 
of behavior which are correlated with delinquency. It may be time to transfer 
'some of the energ^ previously focused on interventions which are now. successfully 
in place to the development of an intensive monitoring strategy for services to 
target students. ' ■ 
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APPENDIX A • 
Intervention Monitoring Documents 

Sample Program, Development Worksheet Strategy Description 
Counseling Intervention 



Strategies and Tasks 



Critical Benchmarks* 



Who will dq what by when? 



3.3 SCS Counseling 

-develop plan for pre- 
senting rap sessions 
(topics., schedule, 
participants) 

-obtain principal 
approval 

-implement and 
monitor 

-establish 'a mechanism 
that will allow SCS 
to intervene early in 
nonat t endanc e /tr uan cy 

cases 

establish a schedule 
* that includes tracking 
attendance, counseling 
returning suspended 
students, and providing 
information to school 
administrators 

- counseling results to 
referring teachers 
via referral form 

- follow-up counseled 
students (teacher 
contacts, selfrimrove- 
ment contracts, parent 
contacts 



At least 3 rap sessions 
per school are held 

Principal agrees to 
procedure which sends 
tardy and truancy cases 
to SCS before the point 
oi suspension 

SCS records indicate N:hat 
counseling sessions were 
held with all students 
identified 



SCS, 11/15 and ongoing 



Program coordinator , 

SCS, and school personnel, 

9/18 



Program coordinator, SCS*, 
9/18 



SCS records indicate 
follow-up contacts 
with students, teachers, 
and parents 



SCS, ongoing* 



SCS , ongoing 



MONTHLY PLANNING CALENDAR FOR November ^ f l 9QI 



Name: : __ School: 

* > 



" S% V^ Activity 


In^ormat ion/ 
Documents to 
be t\irned in 


Expected 
Compl et ion 
Date • 


1 — J 

"Actual 
€ojnpletl on 
Date 


Comments 


Conduct regular 1neet?ings of ^ 
Student Leadership Team, ^ 
Parent Leadership Team, 
SCST • " ■ 


sign-ins , agendas 
minutes 


12/10 . 

r 






Confiuct peer counselor training 
and/or sessions 

* 


sign-ins, 
training 

evaluation survey 


12/10 

5 , 






Submit PATHE Activity reports 
for 

•School pride campaign 
Open house 

Parental involvement 


"reports for each 
activity 
listed 


12/ 10 




• 


* ** \ 

Maintain disc ipl in^#re f er ral 

forms 


completed forms 


12/10 

• 


• 


Collelct completed forms in person from * 
principal, vice principal, guidance counselors 

at least once^a^Week. *Be sure teachers , 
ind administrators have sufficient copies 

of forms. 


*** 

Collect-Suspension information ^ 


copy of school 
Report ^for Novemb 


12/10 






'*** 

Plan to conduct RAP sessions 


plan 


1 * 

12/10 . 




RAP sessions should include an effort to 
modify students 1 antisocial beliefs as 


*** 

Conduct first RAP session " - 


sign- ins 


12/10 ; 




identified toy the student questionnaire. 
Other topics might include student 
victimization and gangs at school. w 



~Q s related to Counseling intervention. ' 
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Intervention 



,.ATHB SCHOOL IMPLEMEHTMIOW REPOKT 
Counsel ing — 



WINGS 
fidelity 2_ 

intensity 

duration 4_ 

support i_ 



3.25 



Intended Activities 



Status and/or 
Date Initiated 



I 
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1W v-lop Plan for preset rap sessions to students. 
,btaii» principal approval. 

r letticfit and monitor. 

^m^i^" - oounsei returning suspe ! 

atlldCfltb. 

H..K>rl of counseling to referring teachers 

vi'i ilifeCiplUM referral form. 

Make i«rwu contacts as needed. 

U.e ,olf improvement contract as needed. 




9/B1 

yes 

yes 



yes 



yes 



Desc 



ription of implemented Activities 



I contacts for RAP sessions' 



25 



— , . ill i conta^a 

contacts for counseling* 3" PA THl><69\ 
students suspended who had contact with PATHB- ^ ^ 



contracts* 0 



parent contacts 



Support •2S ta *' 



WP with ^Ji^'SlSSS ST .mum ■* »■*• - d 



aVso Student Leadership Tea* menbers 
at Rivers Middle ScHpol 



to 



APPENDIX B 
Rating Scales 



Fidelity Scale _. v§ _ 

. - •. 

The rater compares implemented activities and tasks 
as documented on the Implementation Report to the 
standards for implementation described on this and 
subsequent pages. After checking off which of 
the standards have been met, a rating is assigned 
•to each intervention using the following scale: 

4=exceeds standards 

3=meets standards completely or with approved 
s adaptation 

2=meets 50-99% of standards 

l=meets 1-49% of standards 

0=does not meet standards/not implemented 



30 



Intervention 


< 

Standards 


Tutoring 


-outside tutors involved 
-regular schedule of tutoring 

contacts by C^S. 
-target students included 


Counseling 

✓ 


• 

-RAP sessions jplan on file 
-evidence of kSp sessions 
-evidence of at least 10 

parent contacts 
-contact with at least 90% 

of students suspended 
-system established to 

monitor attenuaiice proDiems 
-target students included in 

counseling 


study skills 


-plan on file 

-evidence of implementation 


Job seeking skills 
* 


-distribution of materials 
pre sen tdtions to groups on 
job seeking skills or career 


* 


awareness * " 


Services to target 
students 


-individualized instructional 
plan/needs assessment or 

.problem statement for every 
target student 

-letter sent to parents 

^positive parent contact for 

C VCJL j UaJLUCL OUUUCll U 

*-at least one contact with 

each target student 
: : , rt «t — 



Intervention 



Standards 



Resource room 



-needs assessment memo used 
after each grading period 

-resource room open house 
conducted 

-evidence of teacher use 



Faculty inservices 



-part icpat ion in team build- 
ing conference 

-teacher participation in 
Student Team Learning 
training 

-inservice plan on file which 
addresses identified needs 

-evidence of implementation 
of plan 



School pride 
campaign 



•pilan on file for whole year 
-evidence of implementation 



Curriculum review 
and revision 
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-CTBS analysis on file 
-plan to impr6ve identified 

weaknesses on file 
-evidence of meetings with 

teachers re plains and weak- ' 

nesses 

-failure data collected and 
used to "identify students 
and teachers who need help 

-help provided identified 
teachers and students 

-mini-tests in use 

-instructional materials and 
teaching strategies distri- 
buted 

-assistance in use of curricu- 
lum niiiflo^ 



Intervention 


Standards 


Field trips 


-plan on file 

-one trip for each of three 
areas (career , academic , 
.cultural) 

-scheduled throughout the 
year 

* — — 


Reading Experience 
Program 

\ 


-plan for schooi-wi^e program 
; on file 

-evidence of implem^t^ion 

-motivational activities 
planned and implemented 


Exploratory program 


-schedule on file 
-display of exploratory 
products in school 
-schedule implemented 


Discipline policy 
review and revision 

• 


-$ch6ol and classroom rules 

developed by teachers and 
V students and posted 
'-referral procedures and 
discipline referral form 
approved by principal 
-referral procedure imple- 
mented so that includes 

s « c * s « 

-assi'tance given to teachers 
. with classroom management 
problems (at least 10) 
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Intervention 


Standards 




Intervent ion 


Standards 




\ 


Peer bounseling 


-counselors trained 

-plan on file 

-counseling services 
provided on regular sche- 
dule 




Career exploration 
program 


-studepts recruited to 
attend Trident Technical 
College fall program 

-follow-up meeting with 
students held 




- * 


Extracurricular 
activities 

> 


High School; 
-club plans or charters on 
file 

-club mid-year activity 

reports on file 
Middle School": 
-clubs begun last year 

continue ■ 




Business Education 
Partnership 

r 


-team formed with school 

and buisnfess members 
-P&THE specialist included 

on team 
-jplan developed to improve 

school management 
-implementation of plan 

evident 






Student Leadership 
Team 

) 

J 


-meetings held regularly 
-satisfactory team plan on 
file 

-evidence of plan implemen- 
tation 

-central leadership confer- 
* * ence attended 


r 

> 


Parent Leadership 
Team * f 


-regular meetings held 
-satisfactory team plans on 
file 

-evidence of implementation 

of plans 
-leadership training 

attended 






Student Concerns 
Support Team 

* 


V 

-regular meetings held 
-contracts on file 
-satisfactory team plan 

on file 
-evidence of plan implemen- 

tat ion 
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Curriculum Support 
Team 
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-regular meetings held 

-contracts on file 
• 

-satisfactory team plan 
on file 

-evidence of plan implemen- 
ts i+inn 


• 
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ftiteja&±ty Rating Scale 



The rater compares intensity of progaraflf^ services as documented on the Implementation Report to the standards 
given below and rates each intervenj^ion using the following scale: 



4=exceeds ^stan^atd by more than 5% 
3=meets standard +5% 



2=raeetl 50-95% of standa'rd 
l=meets 5-49% of standard 
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Intervention 


Intensity Standard 


Tutoring , 


200 incidents of tutoring 


Counseling 


300 indidents of counseling 


Study skills ' j- 


50 incideats of partici- 
pation 


Job seeking skills • 


1 all school activity or 
5Q incidents participa~ 
tion . 


Services to target 
students 


3 contacts with every 
target "student 

— 


Resource room and 
services to 
teachers * 


# teacher checkouts 

# teachers 


Faculty inservices 


1 school level, 5 teachers 
attend -S.T.L. , and 5 attend 
team buildinq 


oc noo x pr i de 
campaign 


3 all school activities 


Curriculum review 
and revision 


xii^xuaiiub v~ • o» racuxty 
contact = 2 x # teachers 


Field trip program 


. 1 field trip conducted 


Reading Experience 
Program 


entire school, ntonthly, for 
more^than 15 minutes * , 

• . 



Intervention. 


i ■ — — ■ 
Intensity Standard 


Exploratory program 


entire school, weekly 


Discipline policy 
review artd revision 


# discipline referral 
forms =# suspensions -f 
h # suspensions 


Peer counseling 

V 


related to peer 
counseling 


Extracurricular 
activities 


FT.S., - 10 club plans 
M.S. - 100 incidents 
contact 


Student Leadership 
Team 


30 incidents attendance 


Student Concerns 
Support Team 


20 incidents attendance 


Curriculum Suppoirt 
Team 


20 incidents attendance 


UliCCi CApJLUi. Ct L1UI1 

programs (T.T.C.j 


ou inciu&nts aLLenaance 


Business Education 
Partnership 


3 meetings 


Parent Leadership 
Team 


20 incidents attendance 
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